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ABSTRACT 



Prior evaluation has been concerned with the attitude of 
high school students toward the educational interactive video program for the 
year in which they were participants. This analysis assessed whether there 
had been changes in attitude toward the program from one year (1995) to the 
next (1996) and if improvements made based on the 1995 evaluation had been 
effective. Three hundred forty- four (106 from 1995) high school students 
enrolled in an interactive video class participated in this evaluation. 

Change was assessed on seven constructs and by open-ended questions 
concerning strengths, weaknesses, and suggested improvements. This analysis 
revealed that there has been a significant improvement in equipment and its 
maintenance. Strengths of the interactive video program have remained 
relatively constant over this two-year period. There were no significant 
differences in students' evaluation of the interactive video program over the 
two years, but there was concern over scheduling and behavior weaknesses. 
Several charts illustrate results. (AEF) 
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Abstract 

Prior evaluation has been concerned with the attitude of high school students toward the 
educational interactive video program for the year in which they were participants. This analysis 
assessed whether there had been changes in attitude toward the program from one year (1995) to 
the next (1996) and if improvements made based on the 1995 evaluation had been effective. 
Three hundred forty four (106 from 1995) high school students participated in this evaluation. 
Change was assessed on seven constructs and by open-ended questions concerning strengths, 
weaknesses, and suggested improvements. This initiates a trend analysis to be continued yearly. 
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Prior evaluation with the data used in this study has been concerned with the attitude of 
high school students toward the interactive video program (ITV) for the year in which they were 
participants. The purpose of this analysis is to determine if there have been changes in attitude 
toward the program from one year (1995) to the next (1996). This is the beginning of a trend 
analysis to be continued each year. By comparing attitude changes from year to year, the impact 
of improvements and changes to the program/facilities can be detected. If a supposed 
improvement has had a substantial effect, it is hoped that effect will be detected. If the effect 
reflects improvement in attitude, then it will be assumed the improvement caused this change. 
This would suggest continuation of the improvement. If the effect seems to be detrimental, this 
would suggest the problem should be approached by another method. In addition, this approach 
permits facility administrators to anticipate or detect problem areas while they are still in a 
manageable condition. 

Subjects 

All students enrolled in an interactive video class at this facility during the Spring semester, 
1995 and Spring semester, 1996 were surveyed. Surveys were administered during the regularly 
scheduled class time by the class instructor or remote facilitator. For analysis, respondents were 
categorized by site (home or remote), by year of program existence (year 1 or year 2), and by 
survey year (1995 or 1996). Of the 184 returned student surveys during Spring 1995, 106 were 
completed by high school students. Twenty-one of the 66 home site respondents were 
participating in a first year program. Fifteen of the 40 remote site respondents were participating 
in a first year program (see Figure 1). Surveys were returned from respondents at nine different 
schools in nine subject areas (see Figure 2). 
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There were 238 returned surveys by high school students during Spring 1996. Seventy- 
eight of the 161 home site respondents were in a first year program. Forty-seven of the 77 
remote site respondents were in a first year program. Surveys were returned for 12 subject areas 
(all levels of Spanish were coded Spanish) from 14 high schools. 

All student surveys (344) were entered for analysis. Three hundred nine responses were 
coded as non-applicable. This was less than 5% (7392 responses). It was assumed that those 
who marked non-applicable could not be ranked as undecided since that option was offered and 
was not chosen. Since any numeric value assigned would bias the results (l=strongly agree, ergo 
0 would be very strongly agree) and the proportion was relatively small, these were used as 
missing values. In addition, 91 responses were not marked. These were also used as missing 
values yielding a grand total of 400 missing values (<6%). Although the proportion of missing 
values is relatively small, if listwise deletion were used many cases would be excluded from 
analysis. To prevent this, mean substitution was used for factor analysis. 

Measurement and Analysis 

The survey instrument consisted of demographic information (school, gender, grade, etc.) 
and 34 questions. Question 1 requested reason for taking the ITV class. Questions 2-3 1 were 5- 
point Likert style questions. Questions 32-34 were open-ended requesting strengths, weaknesses, 
and suggestions for improvement for the ITV program. Questions 21-26 were to be answered by 
remote site participants only. Six questions were negatively stated in the questionnaire. These 
were reverse coded for this analysis. Previous exploratory analysis has provided a seven-factor 
model for the questionnaire as well as reliabilities of the factors for the Likert type questions in 
common for both remote and home site students. While these models (1995, 1996) were 
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similar, there were some discrepancies between them This analysis uses one model to conduct 
multivariate analyses. 

Responses to open-ended questions (Q32-34) were initially coded based on the response. 
After determining similarities of the responses, these were placed in categories. For example, the 
suggested improvements response “have schools on the same schedule” was coded as ‘38’. It and 
the responses “synchronize time schedule” and “establish snow schedule” were then summed to a 
major category “scheduling’. If a specific response was infrequent (<10) and could not 
reasonably be included within a category, it was included in other. For this analysis, if either 
survey year (1995, 1996) had sufficient responses to create a category, that category was included 
for contrast even though one year may have zero responses. (Raw responses and their codes are 
listed in Appendix A, Tables 1-3). Categories were then summed to major categories and the 
proportion of categorical responses was contrasted by survey year (1995-1996) for all categories. 
The assumption was made that there would be similar proportions of responses for each category 
for each year. It must be remembered that these are not independent responses. A respondent 
could submit three responses for strengths. All three responses may be variations of items coded 
as Availability. 
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Results 

Strengths 

There were 142 responses concerning strengths (106 respondents) in 1995 and 352 (238 
respondents) in 1996. The proportion of responses citing classes offered at remote sites as a 
strength had decreased (8% vs 1%) from 1995 to 1996. When, however, categories were 
summed (ie, remote and availability became access - meeting people and open-minded became 
interaction - technology and instruction became learning), no practical differences were detected 
(see Figure 3). For example, in 1995, 39% of the respondents cited interaction as a strength. In 
1996, 38% of the respondents cited interaction as a strength. This finding indicates that those 
areas perceived as strengths in 1995 are still perceived as strengths in 1996. 

These categories are further depicted by site (Figure 4) and by existence year (Figure 5). 
Typically remote site students perceive availability or remote as more important (assuming 
number of responses reflects importance) than home site students, while home site students 
perceive meeting people or interaction aspects as more important than remote site respondents 
(see Figure 4). When contrasted by year of program existence, students in the first year of the 
program in 1995 did not differ from their counterparts in 1996 nor did those in the second year 
(see Figure 5). 
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Weaknesses 

There were 128 responses to the weakness question in 1995 and 323 in 1996. The 
proportion of responses citing audio problems had decreased from 35% in 1995 to 16% in 1996. 
The proportion citing video problems had also been reduced from 19% to 2%. Minor equipment 
problems, on the other hand, were not a category in 1995, but were 16% of the responses in 1996. 
The scheduling category had also increased from 2% in 1995 to 12% in 1996. 

When categories were summed (camera, sound, severe equipment problems, and minor 
equipment problems formed equipment - remote and instruction formed instruction) differences 
between 1995 and 1996 were still detected. In 1995, 61% of the weakness responses concerned 
equipment failure or problems. This was reduced to 43% in 1996. The scheduling response 
remained an increase in proportion from 1995 to 1996 (see Figure 6). These responses are 
further depicted by site (see Figure 7) and by year of program existence (see Figure 8). Home 
site responses (Figure 7) and responses by students in an established program (Figure 8) cite 
behavior (cheating, lack of discipline) as weaknesses of the program more frequently than remote 
site or those participating in a new program. 

These results indicate that although there may still be equipment problems that need to be 
resolved, there has been a significant improvement. More troubling is the increased incidence of 
scheduling problems and the non-significant increase in behavior cited as a weakness. While a 
proportional decrease in responses related to equipment requires an increase in another category 
(proportions must sum to 1), 62 weakness responses were related to behavior (cheating, lack of 
discipline, see Table A-2) in 1996 compared to 15 in 1995. In addition, 39 weakness responses 
referred to scheduling problems in 1996 but only 2 in 1995. 
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Suggested Improvements 

There were 93 suggested improvements made by high school students in 1995 and 230 
made in 1996. The larger proportion of suggested improvements in 1995 were concerned with 
audio (20% vs 11%), camera (9% vs 1%), and remote (21% vs 12%) than in 1996. Suggested 
improvements in scheduling (1% vs 12%) and general equipment (2% vs 14%), on the other 
hand, had increased in 1996. When audio, camera, and equipment categories were summed to 
equipment (see Figure 9), although a larger percentage of the responses occurred in 1995 (31%), 
it was very similar to 1996 (26%). The nature of the suggested equipment responses, however, 
had changed. In 1995, the most frequent equipment suggested improvements were ‘fix audio’ 
and ‘fix camera’. In 1996, suggested improvements centered around new equipment (ie, 
chalkboards, individual microphones). 

When summed, suggested improvements in remote and scheduling remained the same. 
No other categories had changed significantly. The suggested improvements remote category 
included responses in 1995 indicating more remote classes should be offered. Obviously, more 
have been made available since that suggested improvement has decreased. 

These responses are further depicted by site (Figure 10) and program existence year 
(Figure 11). These responses and the weakness responses suggest that while all equipment 
problems have not been resolved, there has significant improvement. 
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Factor Comparison 

Factors did not load on the same questions consistently from 1995 to 1996. The Audio 
and Environment factors loaded on the same questions, and Materials Support and ITV 
evaluation factors loaded on similar questions (one question was added to each in 1996). There 
were several discrepancies, however, in the Student Behavior, Class Evaluation, and Interaction 
factors. When the 1996 data was forced to load by the 1995 model, reliabilities did not differ 
appreciably between survey years. When the 1995 data was forced to load by the 1996 model, 
reliabilities again did not differ appreciably between survey years. Some questions, however, did 
not fit either model substantively. For this contrast questions were placed on the factor which 
they appeared to fit logically. Reliability for both groups in this model was then determined (see 
Table 1). Although the reliability of the student behavior factor could have been increased to 
0.62 by combining it with the interaction factor, this was not done. Exploratory analyses from 
both survey years have yielded a seven factor model. To combine two factors would alter that 
model significantly. The questions would also suggest that a separate factor could be established 
distinguishing teacher from class. This also was not done. The new model was an adaptation of 
the two previous models with as little change as possible while still providing a logical fit. Since 
this model fit reliability analyses almost as well as either of the models developed from the 
survey year data and it provided a logical explanation of the factors, it was used to contrast the 
survey years. 

Initially the mean for each respondent on each factor was determined and imputed into 
the data. Multivariate analysis of these factors was then conducted using survey year (1995- 



Comparison 95-96 
21 



site (remote, home), and year of program existence (yearl, year 2) as independent contrasts. Due 
to the relatively small sample size in 1995 (one group would only have 15 members if site and 
year were contrasted together), two analyses were run: one using site with survey year and one 
using year of program existence with survey year. The purpose of these analyses were primarily 
to determine if there had been changes in the interactive video program as students perceived it 
from 1995 to 1996. There were no significant main effects of survey year. Since this program 
had been rated so highly in 1995, any significant difference by survey year would most likely 
indicate a change for the worse (see Figure 12). There was, however, a statistically significant 
interaction effect of survey year with year of program existence (F=3.79‘, df=6,276, p<.001). The 
effect size was very small (0.08). Audio was the only significant contributor (F=5.75, df 1, 281, 
p<.02). This interaction is depicted in Figure 13 . The range of values for audio varied from a 
2.11 (agree) to a 2.52 (agree). Although statistical significance was detected, there appears to be 
little practical importance. 

In addition, there were main effects of both year of program existence and site. These are 
are not reported in this study. 



'All statistics are reported using the square root of the raw scores. Homogeneity of 
variance (BoxM) was not achieved using raw scores but was established with the square root 
function. Means are reported as raw scores. 
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Table 4 

Factor Model for Contrasting Survey Years 1995. 1996 



Factor 


Loading 


Question 

Number 


Label 


Materials Support 


.73 


Q4 


Returned Work 






Q28 


Talk to Teach as needed 






Q29 


Class materials timely 






Q30 


See Materials on System 


ITV Evaluation. 


.79 


Q14 


ITV Good Addition Currie 






Q15 


R-Hestitate Take Anothr ITV 






Q16 


Choice - ITV Class 






Q17 


ITV Good Way Offer Class 






Q18 


Take Another ITV 






Q19 


R-ITV More Difficult 


Audio 


.78 


Q27 


Hear Students other sites 






Q31 


Hear Quest other Sites 


Environment 


.55 


Q2 


Amt Desk Space 






Q3 


Clear sight TV 


Student Behavior 


.53 


Q8 


Behav better ITV 






Qll 


Better Listener 


Class Evaluation 


.67 


Q5 


R- Limit ITV Grade 






Q7 


R Most Talk by Homesite 






Q10 


R More Cheating ITV 






Q12 


Study same ITV 






Q13 


Tchr Attn Same Home/Remot 






Q20 


R-More Study /Prep ITV 


Interaction 


.47 


Q6 


Know Stud Other Schl 


'' 




Q9 


Meet Other Schl Stu mre ofte 
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Summary 

Strengths of the interactive video program have remained relatively constant over this 
two-year period. Interaction with others, access to previously unavailable classes, and learning 
(including technological exposure were most frequently cite in both years. 

Responses to weaknesses of the program have, however, changed indicating significant 
improvement in equipment. This, however, has been replaced with increased citation of 
scheduling and behavior weaknesses. The scheduling weakness is also reflected as an increase in 
suggested improvements. 

There were no significant differences in students evaluation of the interactive video 
program between those participating in 1995 and those participating in 1996. A significant 
interaction effect detected between survey year and year of program existence showed very little 
practical importance. It was, therefore, concluded there was no evidence of change in attitude 
toward or evaluation of the interactive video program by high school students from 1995 to 1996 
on this portion of the questionnaire. There are differences, however, in student evaluations by 
site and by year of program existence. 

Recommendations 

This evaluation indicated there has been a significant improvement in equipment and its 
maintenance. There are, however, other problems that have occurred. Due to the increased 
frequency of response concerning problems with school schedules, this needs to be investigated 
to determine if this problem can be resolved. Students need to be on similar schedules if at all 
possible. In addition, there may be some problem developing in student behavior. Home site 
students have indicated a lack of discipline and/or cheating at the remote site. This may or may 
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not become a problem, but does merit further investigation. 

In addition, the questionnaire needs to be revised. Three factors (Class Evaluation, 
Student Behavior, and Interaction) are not measuring consistently across years. This indicated 
that the questions within these factors are not clear, or the factor needs to be strengthened with 
additional questions. The cautions included concerning student behavior and class evalution my 
also reflect this problem. 
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Appendix A 



Table 1: Responses to Strengths 

Table 2: Responses to Weaknesses 

Table 3: Responses to Suggested Improvements 
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Table 1 

Codes for Answers given in response to the open-ended Strength 



Factor Sub-factor Code Label 



Interaction 


Meeting People 


0 


'Meet more people' 






2 


'Interaction /Oth Schl' 






10 


'Meeting different Schools' 


Interaction 


Open-minded 


21 


'More Open-minded' 






30 


'More Opinions' 


Learning 


Technology 


4 


'Different - Rerun Lecture' 






6 


'Picture' 






7 


'Elmo' 






9 


'Less intimidating’ 






15 


'Technology Exposure' 






17 


'More Interesting' 






24 


'Something New' 






25 


'More I/A Conv' 



Learning 


Instruction 


5 


'Instruction' 






19 


'Better Understanding' 






20 


'More Hands On' 






22 


'Prepare for College' 






23 


'Better Classes' 






26 


'Quality Teacher' 






33 


'Easier' 






34 


'Question as needed' 






36 


'More Learning' 



Access 


Remote 


1 


'Remote' 


Access 


Available Classes 


3 


'More can take' 






8 


'Class Variety' 






11 


'Wider Access' 






18 


'Take Prev. Unavail 1 
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Table 1 (Continued) 



Factor 


Sub-factor 




Code Label 


Behavior 




12 


'Greater Student Responsibility' 






13 


'Less Cheating' 






14 


'Increase Listening' 






29 


'More Participation' 






35 


'Better Study Habits' 


Other 




16 


'None' 






27 


'Saves Money' 






28 


'More students/teacher' 






31 


'Material Timely' 






32 


'Better Environment' 
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Table 2 

Weakness Response Codes Bv Factors and Factor Sub-divided 



Factor 


Sub-factor 


Code 


Label 


Behavior 




0 


'Increased Absence' 






4 


'Easy to Cheat' 






5 


'Noisy (Talking)' 






10 


'Lack Participation' 






21 


'Lack Discipline' 






29 


'Pay less Attention' 






36 


'Boring' 


Equipment 


Sound 


1 


'Equipment - Sound' 






30 


'High Voice not Heard' 


Equipment 


Camera 


2 


'Equipment - Camera' 






14 


'Video Intimidating' 






38 


'Video Small' 






39 


'TV Glare' 


Equipment 




6 


'Time - Iniating Equipment' 






19 


'Lack Communication' 






24 


'Equip Fail - Severe' 






42 


'Slow - Delays' 


Equipment 


Minor 


28 


'Equip Fail - Occas' 


Instruction 




7 


'Attention to Remote Students' 






8 


T-l diff with remote' 






12 


'Teacher Attn to HomeSite' 






13 


'Instruction Light' 






15 


'Teacher attn divided' 






25 


'Little Learning' 






35 


'Difficult Learning' 




\ 


40 


No Extra Help' 
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Table 2 (Continued) 



Factor 


Sub-factor 




Code Label 


Instruction 


Remote 


9 


'Teachers Need to be There' 






11 


'Personal Contact Other Student' 






22 


'No teacher at Remote' 






27 


'Personal Contact Techer' 






31 


'Impersonal - slight' 






34 


'Remote less impt' 


Scheduling 




16 


'Different School Holidays' 






23 


'Schedule Conflicts' 






26 


'Class Cancelled' 


Other 




3 


'3 hour meeting too long' 






17 


'Mail Late' 






18 


'None' 






20 


'Too many students/teacher' 






32 


'Due date materials' 






33 


'More space' 






37 


'Large Classes' 






41 


*No Field Trip' 
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Table 3 

Suggested Improvement Response Codes Bv Factors and Factor Sub-divided 



Factor Sub-factor Code Label 



Equipment Sound 
Equipment Camera 
Equipment 



Learning Class 



Learning Discipline 



Learning Instruction 



Remote 



1 


'Fix Sound' 


2 


'Fix Camera' 


16 


'User-Friendly Equipment' 


17 


'Facilitator initiate equipment pri' 


19 


'Class on Equipment Use' 


29 


'Fix equipment' 


30 


'Trained Eqip/fix standby' 


42 


'Need Chalkboard' 


47 


'Better Communication' 


3 


'More class meetings - less lngth' 


14 


'Require # Students/Site' 


21 


'Reduce lecture time' 


23 


'Longer Class Period' 


24 


'Smaller Classes' 


41 


'Rotate Home/remote' 


8 


'Better Discipline' 


28 


'Better discipline - remote' 


44 


'Screen students' 


10 


'No Learning' 


22 


'Better Teacher Interaction' 


25 


'Better Teachers' 


35 


'Syllabus' 


37 


'Experienced Teachers' 


38 


'Remove facilitators' 


46 


'Tutors' 


15 


'Facilitator at Remote' 


18 


'Teacher needs to visit' 




46 
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Table 3 (Continued) 



Factor Sub-factor Code Label 



Interaction 


9 


'More Interaction' 




13 


'Some Total Class Meetings' 




20 


'Invite More Schools' 




33 


'More people' 


Scheduling 


26 


'Synchronize time frame' 




27 


'Same schedule' 




43 


'Snowday Schedule' 


Variety 


4 


'Advertise' 




5 


'More locations' 




12 


'Offer More Classes' 


Other 


11 


'Quit Using' 




31 


'Internet use' 




32 


'More exposure class content' 




34 


'none' 




45 


'Return work' 


Environment 


6 


'Larger Monitor' 




7 


'Better Setting' 




36 


'More seating' 




39 


'Overhead Speakers' 




40 


'Bigger Desks' 
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